HADOOP-18168. Fix S3A ITestMarkerTool dep. on purged public bucket, introduce PublicDatasetTestUtils#4140
Merged
steveloughran merged 19 commits intoapache:trunkfrom May 3, 2022
Merged
Conversation
Contributor
Author
|
Integration tests passing against |
monthonk
suggested changes
Apr 5, 2022
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/tools/ITestMarkerTool.java
Outdated
Show resolved
Hide resolved
dannycjones
changed the title
dannycjones
changed the title
dannycjones
commented
Apr 6, 2022
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java
Outdated
Show resolved
Hide resolved
Contributor
Author
|
Integration tests not currently passing with this commit, diving deeper here. |
Contributor
Author
|
Tests passing against |
steveloughran
requested changes
Apr 28, 2022
Contributor
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
looks good, only some minor tweaks.
it is going to need a mention in the testing doc though as now there's a new option which needs to be set when testing against non-aws stores
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/PublicDatasetTestUtils.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java
Outdated
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ARequesterPays.java
Outdated
Show resolved
Hide resolved
Contributor
Author
|
Tested latest patch against |
Contributor
Author
|
Thanks @steveloughran, I've added a note to I am hoping that if we get all public datasets into For now, I have not tried to move the CSV Scale tests but I'd like to see them moved in a future patch. |
dannycjones
changed the title
steveloughran
approved these changes
May 3, 2022
Contributor
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
+1 for this; will cherrypick to branch 3.3 too
steveloughran
added a commit
that referenced
this pull request
May 3, 2022
asfgit
pushed a commit
that referenced
this pull request
May 3, 2022
asfgit
pushed a commit
that referenced
this pull request
May 3, 2022
…4140) This moves off use of the purged s3a://landsat-pds bucket, so fixing tests which had started failing. * Adds a new class, PublicDatasetTestUtils to manage the use of public datasets. * The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon HADOOP-14661. Consult the updated test documentation when running against other S3 stores. Contributed by Daniel Carl Jones Change-Id: Ie8585e4d9b67667f8cb80b2970225d79a4f8d257
Contributor
|
merged to trunk, albeit with the commit message partially cut. reverted that and recommitted with the full text. test with branch-3.3 in progress |
asfgit
pushed a commit
that referenced
this pull request
May 3, 2022
…4140) This moves off use of the purged s3a://landsat-pds bucket, so fixing tests which had started failing. * Adds a new class, PublicDatasetTestUtils to manage the use of public datasets. * The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon HADOOP-14661. Consult the updated test documentation when running against other S3 stores. Contributed by Daniel Carl Jones Change-Id: Ie8585e4d9b67667f8cb80b2970225d79a4f8d257
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
This moves off use of the purged s3a://landsat-pds bucket, so fixing tests which had started failing. * Adds a new class, PublicDatasetTestUtils to manage the use of public datasets. * The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon HADOOP-14661. Consult the updated test documentation when running against other S3 stores. Contributed by Daniel Carl Jones
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
This reverts commit 6ab7b72.
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
…pache#4140) This moves off use of the purged s3a://landsat-pds bucket, so fixing tests which had started failing. * Adds a new class, PublicDatasetTestUtils to manage the use of public datasets. * The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon HADOOP-14661. Consult the updated test documentation when running against other S3 stores. Contributed by Daniel Carl Jones Change-Id: Ie8585e4d9b67667f8cb80b2970225d79a4f8d257
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Addressing HADOOP-18168, this change replaces the use of
landsat-pdswith its successor usgs-landsat, another public dataset.landsat-pdsrecently was cleaned up excluding some specific files and is no longer suitable for this test case.usgs-landsatis a requester pays bucket, so costs will not be passed on to the owner of the bucket - instead, they will be paid by the test runner.We introduce a new class
PublicDatasetTestUtilswhich provides a simple interface for getting file systems for a specific purpose. This change allows the bucket used to be replaced or for the tests to be skipped - for example, for S3-compatible stores or non-awspartition S3 endpoints.How was this patch tested?
It will be tested from a
eu-west-1EC2 instance against aeu-west-1bucket.